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ABSTRACT 
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INTRODUCTION 

The decade of the 60 's has witnessed the investment 
of billions of dollars by the federal government and private 
foundations in the attempt to improve education in the 
nation's schools. What effect has this expenditure had on 
the improvement of school progrcims? How wisely have these 
dollars been spent? What factors contribute to effective 
improvement of school programs? These questions, among many 
others, have been asked by the funding agencies; answers 
must be provided. 

The task of answering these questions falls upon 
educational evaluators and researchers. During the past 
decade, the profession has reviewed possible means of 
gathering data bearing on these questions. Existing 
methodologies and goals of evaluation have not been 
adequate for the task and consequently are requiring consid- 
erable re-examination. Scholars from diverse fields have 
tried to conceptualize and implement new approaches. 

Although substantial advances have been made toward the 
delineation of a new evaluation theory and new methodologies, 
much remains to be done before answers can be given to the 
many legitimate questions which have been raised. 

Among the new approaches for coping with the emerging 
problems of evaluation of new curricula and new programs, 
the concept of systems analysis has great appeal to 
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educational planners and evaluators. Systems analysis can 
provide a means for systematizing efforts and conceptualiz- 
ing approaches in the evaluation of program outcomes 
(see Schutz, 1969). While the application of the systems 
analytic paradigm to educational problems involves a con- 
siderable amount of "slippage" when compared with its 
application to engineering problems , the techniques 
provide valuable insights which may assist the educational 
planner and evaluator to do a more adequate job. 

The purpose of this article is to demonstrate the 
capabilities and limitations of systems analysis as applied 
to the development of a program evaluation plan. The 
authors present a brief review of current evaluation theory, 
a description of general systems theory, the development 
.f a general model for educational evaluation, and the 
application of this general model to a proposed evaluation 
plan for a regional educational laboratory, the Eastern 
Regional Institute for Education (ERIE) . 
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; I; SELECTED CONCEPTS IN EVALUATION THEORY 

'i 

f 

Lee J. Cronbach 

Recent developments in evaluation theory have centered 

I .... 

t around the function of evaluation in decision making. An 

^ influential article, "Evaluation for Course Improvement," 

t 

I 

[ by Lee J. Cronbach (1964) formally ushered in this orientation 

\ 

and has had significant impact upon the emerging field of 
evaluation theory. Cronbach viewed evaluation as functioning 
within a decision-making framework, and identified three 
areas of decision making for which evaluation would be 
useful: 1) course improvement, 2) decisions about individuals, 
and 3) decisions related to administrative regulation 
(1964, p. 232). He emphasized that when course improvement 
evaluation is carried out, one is interested in the multi- 
faceted effects which a course has had upon pupils, not just 
upon narrow instructional outcomes. His belief was that the 
greatest service which evaluation could perform was to identify 
areas in which a course could be improved. He stated that, 

"Evaluation, used to improve the course while 
it is still fluid, contributes more to improve- 
ment of education than evaluation used to 
appraise a product already placed on the market." 
(Cronbach, 1964, p. 236) 

In a movement away from the earlier emphasis on terminal 
evaluation, Cronbach further pointed out that evaluation 
should engage in process studies, which center on the inter- 
active events of the classroom. According to Cronbach, the 
• analysis of how a course produces its effects should be 
of equal concern with an analysis of proficiency measures 
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and attitude measures. He also indicated that follow-up 
studies should be given greater consideration in planning 
evaluation studies. A major thrust of Cronbach's article 
was that comparative studies which attempt to compare 
outcomes of one course with another course should not predom- 
inate in evaluation plans. 

Michael Scriven 

In a widely received and acclaimed paper entitled 
"The Methodology of Evaluation" (1967), Michael Scriven 
took issue with what he perceived as Cronbach's lack of 
concern for comparative studies and further extended methodo- 
logical concepts for evaluation. He introduced the 
terminology of formative and summative evaluation, defining 
formative evaluation in much the same way that Cronbach 
talked about evaluation for course improvement. In essence, 
Scriven' s formative evaluation consists of making improve- 
ments in a course or an educational "instrument" (broadly 
defined) while it is in its formative stages of development. 
In contrast with Cronbach's position, Scriven insisted that 
comparative studies or summative evaluation must play a major 
role in educational evaluation. The purpose of summative 
evaluation, in Scriven' s view, would be to provide teachers, 
administrators, and the public with evidence regarding the 
value of a certain instructional procedure or curriculum 
when compared to outcomes achieved by a competing curriculum. 

Ultimately, Cronbach and Scriven were to agree that 
there is need for both types of evaluation; however, Cronbach 
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(1965) insisted that evaluators who conduct formative studies 
to improve a course should not also engage in summative 
studies of the same course because of the diverse nature 
of the two types of evaluation. 

Scriven, in his article, made a number of noteworthy 
distinctions among different types of evaluation. With 
an emphasis similar to that of Cronbach, Scriven termed process 
evaluation as the non-inferential study of what actually trans- 
pires within the dynamics of the classroom; he further defined 
process evaluation as the investigation of causal claims re- 
garding interactive classroom processes and stressed the value 
of this type of study. Scriven made further distinctions 
among intrinsic , mediated , and payoff evaluation . He defined 
intrinsic evaluation as an analysis of the consistency which 
should exist among the content, the goals, the procedures, 
and the outcome measures specified for a program; payof f 
evaluation refers to an analysis of operationally defined 
pupil behavior outcomes. He identified evaluation that would 
relate intrinsic qualities to payoff outcomes as mediated 
evaluation . To conduct a pure payoff evaluation, according 
to Scriven, can be a very costly endeavor because it may tell 
nothing about the processes which are intended to bring about 
those outcomes. Therefore, evaluation should center on the 
full description of the context in which the evaluation takes 
place and the processes which are employed to achieve 
outcomes as well as the outcomes themselves. 
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Robert Stake 

Robert Stake's article, "The Countenance of Educational 

Evaluation” (1967a) added a further dimension to the goals 

and the methodologies of evaluation. Stake characterized 

« 

the two basic acts of evaluation as description and judgment; 
he defined the purpose of evaluation (1967b) as that of 
increasing the rationality of decisions which control the 
inputs and the outputs of educational operations. Evaluators 
are exhorted to describe fully and to judge fully all 
components in any evaluation study. In the gathering of 
descriptive and judgmental data. Stake proposed that the 
evaluator consider three types of data domains: antecedent , 

transaction , and outcome data . Antecedent data are infor- 
mation or conditions which may be related in some way to 
outcomes; that is, the evaluator should consider it his 
responsibility to fully describe and document all the 
environmental conditions and influences which he believes 
may have an effect on any given outcome. With reference 
to transaction data. Stake demanded that the evaluator 
carefully observe and record data emerging from the trans- 
actional and interactional classroom processes. He 
broadened the general concept of outcome data to include 
that which goes beyond immediately evident data and which 
includes future application or transfer long after the 
initial observation. 
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Stake advised that evaluators who are processing 
descriptive data look for contingencies among the intended 
antecedents, transactions and outcomes; he further suggested 
that evaluators look, for congruence between intended and 
observed antecedents, transactions, and outcomes* Stake 
pointed out that evaluators must look for empirical 
contingencies among the observed cuitecedents, transactions, 
and outcomes* In regard to the judgmental aspects of 
evaluation. Stake stated that one can judge the character- 
istics of a program with respect to some absolute standards 
of excellence as reflected by personal judgments of scholars 
or experts in a given field; one can also judge a program 
with respect to a relative standard or comparison between 
the characteristics or the outcomes of alternative programs * 
For example, one could evaluate Science— —A Process Approach 
(AAAS) with respect to scientists' and science educators' 
opinions of what elementary school science curriculum ought 
to be; on the other hand, one might compare the outcomes 
of Science — A Process Approach with other elementary school 
science programs* On the basis of either absolute or 
relative comparisons, one makes judgments* 

Daniel Stufflebeeim 

Daniel Stufflebeam (1966, 1969) has written extensively 
on evaluation as a factor in the decision-making process* 

In 1967, he developed and proposed an evaluation model for 
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use by Title programs under the Elementary and 

Secondary Education Act. The purpose of Stuf f lebeam' s 
model was to provide information to local school districts, 
state departments of education, and federal authorities 
to facilitate the decision-making process. A major 
assumption of his model is that key decision makers at 
each of these three levels require diverse kinds of evalua- 
tive information in order to facilitate the many types of 
decisions which must be made. To serve such decision- 
making functions effectively, evaluative information must 
be valid, reliable, timely and credible. 

Examining the kinds of decisions involved in a typical 
Title III project. Stuff lebeam suggested that needed 
evaluation might be divided into four generalized stages 
which he called Content , Input , Process and Product — 
the components which comprise what he termed the "CIPP 
evaluation model." Context evaluation is primarily concerned 
with antecedent conditions and/or needs which must be 
evaluated in order to plan effectively for any program. 

This aspect of the evaluation model calls for the delineation 
of goals and objectives in relation to the needs of a given 
institution or agency. Input evaluation requires an 
institute or an agency to assess strategies appropriate to 
given program objectives and to identify the available 
resources which might be utilized to achieve the program 
objectives. Decisions at this level of evaluation are 

12 



9 



primarily concerned with the specification of procedures, 
staffing requirements, budgeting and the like. Process 
evaluation is defined by Stufflebeam as feedback information 
to project administrators and others to provide for the 
continuous control and refinement of plans and procedures. 
Another important aspect of process evaluation is the 
identification of potential or actual sources of failure 
in a project and the initiation of remedial action, where 
required . Product evaluation is undertaken to determine 
the outcomes. When considered in relation to context, 
input and process evaluation, product evaluation provides 
the decision maker with the information to determine to 
continue, terminate, modify or refocus a project in part 
or in whole. As part of this model, Stufflebeam provides 
a feedback control loop system for the evaluation .of 
federally supported educational programs by local, state 
and federal agencies. 

In a symposium entitled "The World of Evaluation 
Needs Reshaping" at the 1969 American Educational Research 
Association's convention, Stufflebeam (1969) presented the 
outline of an emergent theory of evaluation currently being 
developed by the Phi Delta Kappa National Study Commission 
on Evaluation. The Commission has been involved in a 
three“year effort to develop a new theory of evaluation. 
Issues to which the Phi Delta Kappa Commission has addressed 
itself include the following: 

13 
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What premises are fundamental to evaluation 
theory? 

How should evaluation be defined? 

What steps are involved in carrying through 
an evaluation? 

What kinds of questions should evaluation 
studies answer? 

What kinds of designs are required to 
answer these questions? 

What criteria are appropriate for judging 
evaluation studies? 

Stufflebeam identified several premises which form 

the foundation for this emergent theory; these premises 

specify aspects of the decision-making process and deal 

with information theory requirements, the specification 

of evaluation strategies in relation to different educational 

settings, and the like. Based on these premises, the 

Commission has defined evaluation as follows: 

"Evaluation is the process of defining, obtaining, 
and using information to judge decision alternatives." 
{Stufflebeam, 1969, p. 2) 

Stufflebeam stressed that evaluation is defined here as 
a continuous process. The first aspect of the definition, 
relative to defining information requirements, essentially 
asks the questions: 

1. Who are the decision makers? 

2. What decisions are to be made? 

3. What alternatives are available? 

What kind of information is important? 
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The processes of attaining and utilizing the infor- 
mation must be cast within the framework of the decision 
maker's questions. Evaluative information must meet the 
scientific criteria which are necessary for all. good 
information, i.e., it must be reliable and valid. The 
Commission added seven utility criteria which evaluative 
information must attain. They are ^^levance, significance, 
scope, credibility, timeliness, pervasiveness and efficiency. 
The Commission has tried to provide an evaluation theory 
and methodology which is scientifically respectable and is 
of utility to practitioners. 

Summary and Conclusions 

The evaluation model to be developed in this article 
will be based upon the following important theoretical 
concepts presented in the preceding review of evaluation 
theory: 

1. Evaluation is considered to be a facili- 
tating factor in the decision-making 
process . 

2. Both formative and summative evaluation 
procedures should be included in any overall 
evaluation plan. 

3. Process and longitudinal studies should 
assume important roles in the planning of 
a comprehensive evaluation plan. 

4. Intrinsic and mediated evaluation studies 
are extremely important adjuncts to 
payoff evaluation. 

Evaluators must take seriously the 
charge to describe fully all antecedent, 
transagtion and outcome data. 

15 
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6. Evaluation data must have practical 

utility as well as scientific credibility 
and must be delivered to decision makers 
in time to serve the judgmental processes 
of evaluation. 

In the final analysis, the authors and others agree 
with Cronbach (1964) that evaluation is, in essence, the 
art of asking good questions. In any evaluative effort, it 
is important to determine what questions are being asked 
and by whom; in addition, it is necessary to determine 
what kind of data are required and in what form in order 
to facilitate the judgment of decision alternatives. The 
effective functioning of any evaluation system will depend, 
ultimately, upon the presentation of accurate, pervasive, 
timely data to those who ask questions and make decisions. 
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II. GENERAL SYSTEMS THEORY AND 
A SYSTEMS ANALYTIC MODEL 

"General Systems Theory seeks to classify systems by 
the way their components are organized (interrelated) and 
to derive the 'laws,' or typical patterns of behavior for 
the different classes of systems singled out by the taxonomy. 
(Rapoport, 1968, p. XVIII). Since the framework of systems 
analysis can be traced back to General Systems Theory 
(Bertalanffy, 1968), certain tenets of this theory are 
reviewed to provide a foundation for building a systems 
analytic model for curriculum or program evaluation. This 
review begins with a definition of "systems" and then 
moves on to such conepts as environment, subsystem, open 
system, feedback, and centralization. 

Systems 

"A system is a set of objects together with 
relationships between the objects and between 
their attributes . " 

(Hall and Fagen, 1968, p. 81) 

The objects are the components of the system. In an 
educational system, the components or the objects would 
include: students, teachers, administrators, instructional 

materials and media, buildings, etc. The attributes are 
the characteristics of the objects in the system. Thus a 
student (object) can be characterized in terms of his 
attributes (interests, I.Q., socio-economic status, 

JL f 



height, etc.) while the instructional materials presented to 
him may be characterized in terms of their attributes 
(level of difficulty, type of media required, length of 
study time and so on) • The re lationships (among the 
attributes of the various objects within a system tie 
that system into a functioning whole which is characterized 
by a distinct organization. These relationships are deter- 
mined by the specific problem being investigated, but they 
can be abstracted for an education system in general. For 
example, when organizing for instruction, a teacher may 
select one teaching strategy to be used with students 
with certain specified attributes or characteristics while 
employing a different strategy for other pupils. Thus 
the teacher's instructional plan relates the attributes 
of the two objects, pupils and strategies, in a distinct 
organizational pattern. 

Implicit in the definition of system is the notion 
that "...a system has properties, functions, or purposes 
distinct from its objects , relationships , and attributes . 
(Hall and Fagen, 1968, p. 81). For example, two school 
^j_ 3 't]^icts could conceivesdDly organize instructional programs 
to serve entirely different ends. While the objects, 
attributes, and relationships might be essentially identical, 
the goals of the programs could be distinctly different. 

What is essential to note is that a system has both 
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organization or structure and functions or goals which are 
distinct from one another. To characterize a system, both 
of these notions must be taken into account. 

One issue must be considered before moving on to other 
concepts of the General Systems Theory. The term system 
connotes to many people an abstract mathematical model 
requiring a precision which educational measurement cannot 
provide. This view represents the conceptual approach to 
systems characterized by the work of Ashby (1960) . However, 
a system can also be built upon the phenomena of the empirical 
world. In this approach one would describe the objects and 
attempt to describe the relationships that exist among their 
attributes (e.g., Bertalanffy, 1952). In fact, the capability 
of system theory to characterize the empirical organization 
of components is one of its outstanding features. 

In the preceding definition of a system, the emphasis 
is on the specification of objects, attributes and relation- 
ships of the system (organization) , as differentiated from 
its function or purpose. It should be pointed out that 
for curriculum or program evaluation, the central ''objects'* 
of the educational system arc the student and the curriculum. 
This system shall be called the central subsystem . However, 
both the teachers and, less directly, the administrator 
also affect the student's behavior and attitudes. A 
question then arises as to whether or not to consider these 
individuals as the "objects" of the central subsystem as well. 
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This potential ambiguity can be cleared up by introducing 
a second concept of General Systems Theory, environment. 



Environment and Subsystems 

"For a given system, the environment is the set 
of all objects; a change in whose attributes 
affect the system and also those objects whose 
attributes are changed by the behavior of the 
system" 

(Hall and Fagen, 1968, p. 83) 

Thus the environment — teachers and administrators — 

interact with the student not in and for themselves but, 

rather for the student. In short, the primary aim of 

the educational system as a whole is to increase the 

capabilities of student behavior and to change student 

behavior. Therefore, teachers and administrators can be 

considered the environment for the central subsystem and, 

more specifically, can be considered subsystems . 

"Objects belonging to one subsystem may well 
be considered as part of the environment of 
another subsystem. Consideration of a subsystem, 
of course, entails a new set of relationships 
in general. The behavior of the subsystem 
might not be completely analogous with that of 
the original system. Some authors refer to the 
property hierarchical order of systems ; this 
is simply the idea expressed above regarding 
the partition of systems into subsystems . 

(Hall and Fagen, 1968, p. 84) 

In general, an "instructional system" would be 
comprised of three subsystems: the central subsystem, the 

reference subsystem, and the support subsystem. The central 
subsystem in curriculum or program evaluation would consist 
of the students and the curriculum. The reference subsystem 
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(e.g., the teacher) interacts or interfaces directly with 
the central subsystem, and thus exerts the greatest 
influence on the central subsystem of any environmental 
factor. The support subsystem (e.g. , administrators, 
school board) exerts an indirect influence on the central 
subsystem by directly influencing the reference subsystem. 

Thus far, an educational progrcun may be characterized 
in terms of its components, the characteristics of its 
components, and the interrelations of its components. 
Teacher and administrator components may be seen as 
"environment" to the student component; and, consequently, 
an educational program or system may be considered a group 
of subsystems hierarchically arranged. In order to explain 
the way in which this hierarchical relationship operates, 
some additional concepts are introduced: 



Open Versus Closed System 

"...An open system will attain a steady state in 
which its composition remains constant, but in 
constrast to conventional equilibria, this con- 
stance is maintained in a continuous exchange and 
flow of material. The steady state of open 
systems is characterized by the principle of equi- 
finality; that is, in contrast to equilibrium 
states in closed systems which are determined by 
initial conditions, the open system may attain a 
time independent state independent of initial 
conditions . " 

(Bertalanf fy , 1968, p. 18) 

The key to understanding an open system is the concept 
of equifinality. The open system maintains balance by the 
assimilation of new conditions rather than having to 
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return to its beginning state to achieve equilibrium. 

This concept can be illustrated when one considers a living 
organism as characteristic of an open system; balance is 
achieved as the organism adapts to changing conditions in 
the environment although the initial state of the organism 
never occurs again. The open system, then, tends to increase 
its complexity and order while still achieving equilibrium. 

By contrast, the closed system maintains equilibrium with 
the initial conditions and the general course of events 
is toward leveling down differences and states of disorder. 
The description of an open system most clearly relates to 
curriculum or program improvement or what has been termed 
"formative evaluation." For example, as curriculum is 
introduced into a school, it will undoubtedly undergo 
formative evaluation; and as a result, changes will be 
incorporated; the final curriculum product will undoubtedly 
be somewhat different from the product which entered the 
school. 

Systems with Feedback 

"Certain systems have the property that a portion 
of their outputs or behavior is fed back to the 
input to affect succeeding outputs." 

(Hall and Fagen , 1968, p. 87) 

In the example given above, the feedback mechanism 
enables the system to change continuously during evaluation 
and enables the authors to characterize formative evaluation 
as an open system. 
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Centralization 



"A centralized system is one in which one 

element or subsystem plays a major role in 

the operation of the system." 

(Hall and Fagen, 1968, p. 86) 

With reference to an educational program, the student 
may be considered as the central subsystem as described 
above. With reference to evaluation of a curriculum, the 
curriculum itself would be considered central along with 
the student. 

To summarize the elements of General Systems Theory 
which have implication for the evaluation model to be 
developed, the following should be noted: 

1. The system will be an open system which 
utilizes feedback to insure the continued 
improvement of conditions which will tend 
to maximize the intended outcomes. 

2. The system will be centralized in that one 
subsystem will play a major role with other 
subsystems interacting or supporting the 
elements of the major subsystem. 
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III. FRAMEWORK FOR A SYSTEMS ANALYTIC MODEL 

( 

The tenets of General System Theory, described briefly 
above, serve as ground rules for constructing a systems 
analytic model for curriculum or program evaluation. In 
fact, the descriptions presented above serve as the elements 
put together below* 

The Skeletal Model 

The system comprises the "processes" through which 
any person (or thing) entering the system must pass and 
exit when outcomes have been achieved. Thus a systems 
analytic model requires identification of inputs, processes, 
and outputs. In addition, the concept of environment is 
introduced to explain the hierarchical relation of influences 
acting directly and indirectly upon the central subsystem. 
Three types of subsystems are identified to characterize 
a. system: central, reference, and support. These provide 

the basis for the model shown in Figure 1. The solid arrows 
show the direction of relationships between flow through 
the subsystems and flow between subsystems. The broken 
lines show the feedback throughout the entire system. 

This skeletal model suggests analytic procedures for 
an evaluation program. The first procedure is to determine 
precisely what is to enter the system at all levels. If 
a curriculum is being evaluated with certain students, 
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SUPPORT SUBSYSTEM 



both the curriculum and the students must be described in 
full. In addition, the model indicates that the teachers 
(reference subsystem) and administrators (support subsystem) 
must be considered as part of the entire evaluation program. 
Given the inputs to the system, the processes through which 
the inputs pass must be identified specifically . It is 
entirely possible to take the inputs and to design alternative 
processes for evaluating specified outputs. This would also 
suggest identification of alternative instructional methods 
for evaluative research (see Suchman, 1967) . The output 
section makes explicit every type of outcome to be realized 
by the system. For education, specification of output in 
terms of performance criteria is necessary but not sufficient. 
Any behavior, whether or not it is measurable, should be 
indicated if considered relevant. 

A skeletal model of systems analysis has been presented, 
and examples have been drawn from an abstract educational 
system. The next step is to consider additional elements 
of evaluation more thoroughly in order to develop this 
skeleton model into a generalized systems analytic model 
for curriculum and program evaluation. 

Levels of Decision Making 

Evaluation has been presented as the process of defining, 
obtaining, and using information to judge decision alterna- 
tives. A system has been described as a set of objects and 



23 



the relationships among the objects and their attributes; 
a system has been characterized as an open system with 
feedback comprised of central, reference, and support 
subsystems with inputs, processes, and outputs specified 
for each subsystem. Before demonstrating the application 
of such an evaluation system, it is necessary to introduce 
a further consideration — levels of decision making within 
an evaluative framework. 

When constructing an evaluation plan or implementing 
an evaluation system, it is important to determine the 
different sources and perspectives from which questions 
regarding evaluation might arise and for which answers 
must be provided to facilitate the decision-making process 
(Forehand, 1968). In asking the question, "From whose 
point of view is evaluative data collected?”. Forehand 
distinguishes between two kinds of evaluation — project 
evaluation and institutional evaluation . This distinction 
is useful when one considers the evaluation needs within 
a complex organization engaged in many activities, sub- 
programs, or projects. Project evaluation and institutional 
evaluation are not incompatible; however. Forehand stresses 
that different perspectives might demand important differences 
in the way data may be collected and reported. 

The primary difference between project and institu- 
tional evaluation, in Forehand's terms, is that the 
institution considers the achievement of any particular 
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program as a sub-set in relation to the network of other 
programs and other goals within the institution. Therefore, 
the perspective of the institutional evaluator will be 
quite different from that of the program or the project 
evaluator. The project evaluator would be primarily 
interested in improving output of a single unit within the 
institute. The institutional evaluator, however, centers 
on the study of the institution ' s overall efforts to 
achieve its objectives. A good illustration of the 
difference between project and institutional evaluation 
becomes evident when one considers the effort and evaluation 
needs of a curriculum development center or a regional 
educational laboratory. Such agencies usually have a 
well-defined set of global objectives or a mission to 
fulfill. In order to realize their objectives, these 
institutions typically create sub-divisions to achieve 
certain specific segments of the mission. Each sub-division 
or project must then establish a more specific set of 
objectives and procedures to achieve its goals. The goals 
for the sub-division or project become a specific sub-set 
of the more global goals of the institution. 

The notion of levels of perspective or levels of 
decision making is extremely important in the development 
of an evaluation plan. The distinction between the 
perspective of a project evaluator and an institutional 



evaluator leads to the necessity for different types of 
data collection and reporting. A project evaluator, for 
example, is primarily concerned with formative evaluation; 
consequently, he will generally need micro data in answer 
to very specific questions which will facilitate the 
improvement of programs. The institutional evaluator, on 
the other hand, deals with a multitude of programs; his 
needs call for more generalized or macro data to be used 
in assessing the progress of the entire institution in 
meeting its goals. At times, evaluative personnel at 
both levels will require both macro and micro data. In 
brief, the level of perspective or decision making will 
have an important influence on the identification of the 
components of each of the subsystems . Examples provided 
later in the article will illustrate this point. 

The Generalized Model 

Figure 2 is an expanded version of the skeletal model 
previously presented. The general type of information for 
each subsystem has been identified along with the flow of 
data through the system. This figure also indicates that 
pupil outcomes will generally be of prime concern in most 
evaluation studies. 

For the central subsystem, the background, aptitudes, 
and needs of the students in the evaluation need to be 
considered in setting the objectives for the evaluative 
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program. Furthermore, the content, philosophy, and structure 
(attributes) of the curriculum being introduced need to be 
specified. The model presented in Figure 2 implies that 
it is essential to use the appropriate student population 
for evaluating a curriculum. Once this match has been 
made, the specific components and alternatives will be 
reflected in the evaluation. The output for the central 
subsystem is the behaviors and attitudes of the students 
against which the objectives of the curriculum may be 
evaluated. Based on this evaluation, results in terms 
of revisions are fed back into the central subsystem. 

The inputs for the reference subsystem are concentrated 
on the experiences and aptitudes of the teachers and 
the types of instructional materials and strategies required 
by the curriculum. The processes for the reference system 
are, first of all, the component acts of teaching. In 
addition, the model recognizes that inservice training of 
teachers is essential when a new curriculum is introduced. 
Thus, the components of this inservice training program 
need to be made explicit. The consequent behaviors, skills, 
abilities, and attitudes of the teachers represent the 
output of the reference subsystem. This output serves as 
an input into the central subsystem. 

Finally, the experiences, abilities, attitudes, needs, 
and objectives of each of the components of the support 
system need to be identified. Support personnel, facilities. 



28 



and funding play an important, though indirect, role in 
producing the outcomes desired. The primary responsi- 
bility of administrators, board members, and the local 
school district is to make decisions affecting the 
curriculum, the students, and the teachers. The outputs 
of the support subsystem, in this case, are the decisions; 
these decisions influence the central subsystem by 
inputting into the reference subsystem. 

A Note of Caution 

Several general comments regarding the application of 
a systems analytic model to educational evaluation are 
in order. First, it should be clearly understood that 
the systems analytic model presented is, at best, a 
heuristic device for program or curriculum evaluation; 
it is not a predictive model in the statistical sense. 

This means that in the development of this model, relation' 
ships, objects, attributes, and goals are not necessarily 
quantifiable — given the present state-of-the-art of 
educational and psychological measurement. Thus the 
model encourages the evaluator to consider all relevant 
information for decision making, whether or not it can be 
reduced to a specific quantity. Secondly, this is an 
"empirical'' model. This means that it attempts to 
describe the real world as it exists. The first 
consequence of this is that the model is appliccdale to 
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many different problems. For example, the model's 
components will vary according to the level of information 
required in curriculum evaluation. A second consequence 
of the empirical model is that it may indicate areas 
to be considered that would not have been perceived in the 
real world by a less rigorous approach. However, the 
systems analytic approach is not a panacea for evaluation 
problems; it is an organizational framework which makes 
explicit the nature and relationships of inputs, processes, 
and outputs of a program. A human being with all of his 
capabilities and limitations must apply this discipline 
to a problem. At least this method will make explicit 
what is not being included in a program as well as what is 
included. 
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IV. APPLICATION OF THE MODEL 
TO A SPECIFIC PROBLEM 

In a decision-making framework, the evaluator's task 
in the development of an evaluation system is to determine 
the levels of decision making within and related to the 
educational institution. It is then necessary to determine 
the types of questions which are being asked in order to 
formulate decision alternatives at the varj.ous levels to 
determine what kinds of information are needed, and when 
data are needed to facilitate the decision-making function. 

To illustrate this process of developing an evaluation 
system, the Eastern Regional Institute for Education (ERIE) , 
a regional laboratory funded by the U.S. Office of Education, 
is used as an example. 







An Evaluation System for the Eastern Regional 
Institute for Education 

Briefly stated, the mission of ERIE is to improve 
process-oriented education in the elementary schools of 
the nation. Process education provides more effective 
curricula in such areas as reading, mathematics, science, and 
social studies. A command of basic skills, the development 
of thinking ability, and the tools to continue life-long 
learning are important outcomes of process education. 

To illustrate the development of an evaluation system 
for ERIE, three different levels of decision making within 
or related to the mission of ERIE have been identified. 
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These levels are termed; program level, institutional 
leadership level, and extra-institutional level. 

The program level of operation within ERIE'S structure 
is charged with the responsibility of testing process- 
promoting curricula in "laboratory- type" schools, adding 
needed elements such as adequate objectives and pupil 
assessment devices, and verifying that each curriculum 
produces its intended results . Being satisfied with 
results in a "laboratory-type" school, ERIE then installs 
each curriculum in a network of pilot and demonstration 
schools of diverse characteristics. When installing a 
new curriculum in pilot and demonstration schools, ERIE 
studies factors which facilitate or impede the successful 
implementation of the curricula. 

The institutional leadership level of the organization 
is comprised of the executive officers of the Institute 
and the Board of Directors; the latter group is the policy- 
making body. The leadership level is responsible for the 
formulation and ultimate implementation of the policies 
of the Institute with regard to both programs and personnel . 

Related to the effective operation of ERIE are two 
gj-oups which guide, support and collaborate with the 
Institute in its efforts to improve process-oriented 
education. The U.S. Office of Education, and affiliates, 
comprise what will be termed, for the purposes of evaluation. 
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the extra-institutional level of decision making. The 
United States Office of Education, through its Division of 
Educational Laboratories, reviews and evaluates the general 
operation of the Institute and provides the major source 
of funding. The affilitates include the state departments 
of education, teacher training institutions, and local 
school districts within the region which collaborate with 
ERIE and support its work. 

The distinctions made among the three levels are 
critical for evaluation purposes; each group brings a 
different perspective from which to view the efforts of 
the Institute. These different perspectives determine 
the kinds of questions which each group asks , the type 
of information sought, and most importantly, the types 
of decisions which each group will make. 

Types of Decisions and General Kinds of Questions 
Asked by the Three Levels of Decision Making 

With respect to ERIE'S program level-of-operation, 
component directors and their support personnel are con- 
cerned with the application of criteria for the selection 
of process-oriented curricula and with various program 
outcomes. Decisions to modify programs in various stages 
of implementation are made to make more definitive 
judgments and recommendations regarding program outcomes 
or installation strategies. Questions asked at this level 
of organization relate to the full description of the 
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relationships among both the intended and observed inputs 
into the program and processes employed to achieve outcomes. 
Having determined the what and the how of outcomes , program 
directors and their assistants will also become concerned 
with questions related to why these outcomes were achieved 
with a given population. 

Questions asked by the institutional leadership, both 
executive and policy, generally relate to the effectiveness 
of individual programs and the accomplishments of the 
Institute in general; these questions are posed regarding 
the allocation of resources and personnel and to decide 
to modify, refocus, extend, initiate, or terminate individual 
programs. 

The parties which comprise the extra-institutional 
groups of ERIE must be considered separately, since each 
group makes a different type of decision and, consequently, 
asks a different kind of question. 

The United States Office of Education, as represented 
by the Division of Educational Laboratories, makes 
decisions about the funding which ERIE will receive. 
Consequently, the questions asked at this level relate to 
the efficiency and effectiveness of the Institute in 
relation to its identified mission; in making judgments 
about ERIE, the priority needs of education in the country 
at. large must be taken into consideration. Part of the 
data for decision making for tdie Office of Education comes 
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from site visitors who are selected to review the efforts 
I of the Institute. The questions asked by site visitors 

I may range from the broad type of question similar to 

I those asked by the Board of Directors to very specific 

|t 

( questions characteristic of the program level of operation. 

The extra-institutional group of affiliates is composed 
of three distinct sub-groups: state departments of education# 

teacher training institutions# and local school districts; 
each of these groups makes a decision to collaborate with 
the Institute in the installation of process promoting 
elementary school programs. Each of these groups asks 
questions related to the benefits to be derived from their 
participation with ERIE# and# on the basis of the answer, 
determines whether to collaborate with ERIE or not. For 
example# a state department of education would question 
the effect to which collaboration with ERIE will further 
the objectives which the state holds for its elementary 
schools. Likewise# teacher training institutions ana the 
professors involved in collaborative efforts would ask 
to what extent collaboration would further their own ends. 
Local school districts would question the degree to which 
collaboration with ERIE has a salutary effect on pupils, 
teachers# administrators, program# and the community at 
2_3rge . It is evident that collaboration means different 
things when viewed from the perspective of each of the 
three affiliated groups listed above. While the question 
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asked by each group may be similarly phrased, the answers 
required are quite different. 

Table I presents a summary of the levels of decision 

making. 

Levels of Data 

A brief review, of the terms will prepare the 
reader to analyze the cells in Table II . The central 
subsystem specifies the subjects of interest in an 
evaluative investigation. The reference subsystem is 
comprised of those personnel or materials, instruments, 
etc. which direct];/ interact with the central subsystem 
to produce the outcomes. The support subsystem specifies 
all the indirect conditions in the environment which are 
necessary to bring about the interaction of the reference 
and the central subsystems. By inputs we mean all those 
factors which may influence outcomes. Processes are those 
encounters or interactions which are the vehicles for 
producing the desired outcomes. The outcomes are those 
desired behaviors or expectancies which are the object 
of the entire effort. 

Having identified a level of decision making and the 
0 valuative questions being asked, the next step in the 
formulation of an evaluation plan is to relate the 
decisions and questions to the systems analytic model 
presented in Figure 2. This calls for the designation 
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of components of each subsystem in terms of inputs, 
processes, and outputs. Designating the descriptive data 
to be collected allows the evaluator and the decision 
maker to gain a comprehensive view of the scope of the 
©valuation needs in order to judge decision alternatives 
relative to a given curriculum or program. 

Table II indicates the nine data cells ufsed in ERIE'S 
systems analytic approach to evaluation. Of prime concern 
will be the data in cell 3; the outcome data with the 
subjects of central concern, when interpreted in relation 
with other outcome data provide the substance for formative 
evaluation. 



TABLE II 

DATA CELLS FOR ERIE'S EVALUATION SYSTEM 

Inputs Processes Outputs 

Central Subsystem 1 2 3 

Reference Subsystem 4 5 6 

Support Subsystem 78 9 



Before proceeding further to give an example which 
would illustrate the application of the evaluation system 
to a specific ERIE program, it is of the utmost importance 
to recognize that with respect to the information to be 
gathered and the decisions to be made, the designation of 
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central subsystem will change depending upon the questions 
being asked, by whom, at what level of perspective and 
decision making within or related to the Institute. 

Depending upon the answers to these questions, the central 
subsystem will be any one of the following; teachers, 
administrators, college professors, program components, or 
the Institute itself. 

In Table III which follows, the major components of 
the central subsystem are the student and the instructional 
materials; the questions center on the extent to whi ,h the 
students achieve the cognitive and affective outcomes of the 
elementary school program. Science — A Pr ocess Approach. 

The components of the reference subsystem, in this case, 
would be the teachers and the classroom environment; the 
criteria for selecting these components require that they 
directly interact with the students in the learning process 
to produce the intended outcomes . The components of the 
support subsystem would include other physical facilities and 
support personnel, the administrative support with the school, 
financial support, and the consultant services provided 
by ERIE through the Regional Action Network (RAN) of college 
pj^Qf0ssors. The criteria for identifying the components 
for the support subsystem require that they directly 
interact with the components of the reference subsystem and 
indirectly affect the central subsystem components. 
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AS illustrated in Table IV, teachers may be considered 
the central subsystem when considering questions about 
teacher effectiveness or teacher training in the evaluation of 
Science~~A Process Approach . In this case, the reference 
subsystem would be the Regional Action Network of professor- 
consultants designated by ERIE to conduct the continuing 
inservice training of teachers. The support subsystem would 
include the facilities and resources designated by ERIE to 
conduct the workshops for the training of consultants and 
teachers. In this case the output of central interest 
includes the understandings, behaviors, and attitudes 
produced in teachers as a result of the training program. 

In the example in which teachers are considered as 
the central subsystem (Table IV) , one can identify three 
major direct influences on their behavior: students, 

consultants, and instructional materials. When teachers are 
the central focus for investigation, the question arises as 
to whether or not the students should be included as 
part of the reference subsystem. In Table III, when 
students are considered the central subsystem, the in- 
fluence of pupil behaviors on teacher behavior is accounted 
for by means of the feedback loop. The existence of this 
data as part of an overall evaluation effort and the 
portrayal of student/ teacher interaction in this manner 
preserve the "transaction” aspect of teaching and learning. 
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Consequently r there is no need to include pupils as 
components of the reference subsystem. 

In instances where teacher training is of central impor- 
tance and pupil behavior is only incidental to overall program 
evaluation# students would have to be considered as com- 
ponents of the reference subsystem. This would generally 
be the case when one is solely concerned with teacher 
behavior outcomes as might be the case in the evaluation of 
teacher training programs. 

Up to this point we have demonstrated the applicability 
of the model to micro data gathering and to decision making 
at the program level of operation. The authors contend that 
the evaluation system will prove to be effective at any 
level of decision making. A final example portraying the 
flexibility of the model at a more macro level will be 
illustrated briefly as follows: 

As one views ERIE from the perspective of extra- 
institutional groups# the Institute itself — its programs 
and personnel— becomes the central subsystem about which 
questions are to be asked and data are to be gathered in 
order that these groups may make decisions appropriate 
to their responsibilities. For example# a site visitor 
from the Office of Education might ask the question# "What 
impact is ERIE having upon its region?" In this instance# 
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the Institute and its progrcims become the central subsystem, 
the institutional organizational structure and its internal 
and external communication systems become the reference 

subsystem, and the various sources of funding and ancillary 

✓ 

facilities become the support subsystem. Table V above 
provides the reader with general statement of the primary 
central, reference, and support subsystems from the per- 
spective of each of the various levels of decision making. 
The elements of this Table V indicate the practical utility 
and the flexibility of the model for use in empirical and 
non-empirical evaluation studies. 
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V. LIMITATIONS OF SYSTEMS ANALYSIS 



By this time the capabilities of a systems analytic 
approach to educational evaluation should be evident. 

There are/ nevertheless/ a number of limitations/ or at 
least unfulfilled expectations/ inherent in this approach. 

Aerospace and other defense industries have developed 
systems analysis primarily for application to physical 
systems. Educators tend to think in terms of a direct 
analogy between the application of systems auialysis in 
engineering and its application in education. Consequently, 
many educators reject this approach as "dehumanizing" since 
so much of education cannot be quantified in the manner 
suggested by the engineering model . One limitation of the 
model presented in this paper is that it is not predictive 
in tfie engineering sense. The data obtained from educational 
and psychological measurement is not sufficiently precise or 
sufficiently complete to permit building a predictive model 
for selection among decision alternatives; in addition/ 
much important educational data is non-quantififc;ble . However/ 
this limitation does not mean that systems analysis cannot 
be applied to educational evaluation. What it does suggest, 
however, is that whenever possible, quantitative data should 
be collected; when this is not possible, qualitative data 
should be obtained. Therefore, the model is not limited to 
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behavioral (Tyler, 1950) or instructional (Mager, 1965) 
objectives; it also utilizes what Eisner (1969) terms 
"expressive" objectives. 

This leads to the question: "What will systems analysis 
do for us?" The answer is that systems analysis will enable 
the evaluator to do a more comprehensive job of planni?)^ 
his evaluation effort. Systems analysis applied to educa- 
tional evaluation is a heuristic device for organizing the 
problem in terms of its components and its relationships. 

As such, it reduces the possibility of omitting the collection 
of important information, and it forces the evaluator to 
consider all levels of information required of the evaluation 
program. Finally, it demands that the evaluation design 
make explicit what will be gained from the evaluation, and 
it assures that relevant information will be provided to 
decision makers. Once an evaluation plan is organized, the 
question of measurement arises. Systems analysis makes 
explicit the nature of the data to be collected but systems 
analysis does not tell the evaluator how to measure the 
educational outcomes specified; decisions related to instru- 
mentation are beyond the scope of this article. By using 
this approach, the evaluator can be fairly sure that he has 
identified what to measure in order to provide information 
for the various levels of decision making. 
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SUMMARY 

A system? analytic framework has been developed and 
applied to an evaluation system. The authors contend that 
this approach may provide a useful device for the planning 
and implementation of evaluation studies centered on a 
decision-making basis. Some of the basic notions of emerging 
evaluation theory and basic tenets of General Systems Theory 
were provided as the basis for the development of an 
evaluation system. An illustration of the application of the 
evaluation system was presented using the program plan of 
the Eastern Regional Institute for Education (ERIE) . | 

It has been pointed out that systems analysis applied 
to evaluation provides, at best, a heuristic device for 

f 

organizing, implementing and interpreting evaluation 
efforts. The practical utility of the proposed evaluation 
systems needs to be ascertained; it has proved to be helpful 
to the authors in construction of evaluation plans. To the 
extent that it assists evaluators to identify, collect, 
interpret, and report information of practical utility 
and scientific credibility to decision makers, it will 
prove to be a valid technique. 
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